Skip to content

feat(cuda-device): add SharedPtr, a CTA-shared pointer typed addrspace(3) - #1341

Closed
lucifer1004 wants to merge 1 commit into
NVIDIA:mainfrom
lucifer1004:feat/shared-ptr-1337
Closed

lucifer1004 wants to merge 1 commit into
NVIDIA:mainfrom
lucifer1004:feat/shared-ptr-1337

Conversation

@lucifer1004

Copy link
Copy Markdown
Contributor

Part of #1337; groundwork for #1238.

What this adds

SharedPtr<T>: a raw pointer into the issuing CTA's shared memory (.shared::cta) that carries its address space in its type.

A *mut T in device code is a generic pointer. Once it is passed to a helper, nothing tells the compiler that it points into this CTA's shared memory; helpers are compiled as separate functions whatever their inline attribute. SharedPtr<T> lowers to an addrspace(3) pointer wherever it goes: parameters and returns of non-inlined helpers, struct fields, generic functions and closures. Instructions that take it can use their .shared::cta forms without converting a generic address, which is what #1238 needs for TMA on SM12x (see #1337).

static mut TILE: SharedArray<f32, 256> = SharedArray::UNINIT;

#[inline(never)]
unsafe fn fill(tile: SharedPtr<f32>, i: usize, value: f32) {
    unsafe { tile.add(i).as_ptr().write(value) };   // `tile` is addrspace(3) here
}

let tile = SharedArray::shared_ptr(&raw mut TILE);
unsafe { fill(tile, thread::threadIdx_x() as usize, 1.0) };
API
SharedArray::shared_ptr(&raw mut S) the array's shared address, with no conversion
DynamicSharedArray::<T>::shared_ptr() / shared_ptr_at(bytes) as get() / offset(bytes)
unsafe SharedPtr::from_raw(p) the one explicit locality assertion, like CUDA C++ __cvta_generic_to_shared or CuTe make_smem_ptr
add, offset, byte_add, cast pointer arithmetic that keeps the type
as_ptr() the generic *mut T, for loads, stores and raw-pointer APIs
addr_u32() the 32-bit .shared::cta address, for the _shared_u32 forms (#1236)

Only a mutable form is provided: shared memory is written through the same pointers it is read through, and as_ptr() gives a *mut T that coerces to *const T where an API wants one.

How it works

  • SharedPtr<T> is #[repr(transparent)] struct SharedPtr<T> { ptr: *mut SharedSlot<T> }, with a #[doc(hidden)] #[repr(transparent)] SharedSlot<T>(T) pointee.
  • The importer types a pointer to SharedSlot<T> as an addrspace(3) pointer to T, as it already does for pointers to SharedArray and Barrier. It lowers SharedPtr<T> itself to that same pointer, as a scalar-lowered newtype like ThreadIndex. Construction therefore reuses the existing newtype aggregate path, and reading the field is the existing scalar extract_field.
  • One importer change beyond the type mapping: a Field(0) projection on a SharedPtr place is the identity, so the place walkers drop it. Without that, a SharedPtr nested in another place, such as region.base.ptr after rustc inlines cast, was read as the address of an aggregate.
  • The methods are ordinary #[inline(always)] Rust: casts between *mut SharedSlot<_> stay in addrspace(3), as_ptr is the 3 → 0 addrspacecast that &raw mut S as *mut T already emits, and from_raw is the 0 → 3 cast.
  • SharedPtr cannot be a kernel parameter. The exporter already rejects addrspace(3) entry parameters, and it does so for SharedPtr too: kernel ... parameter 0 is a pointer into shared memory (address space 3) ....
  • Pointers from map_shared_rank, which may name another CTA's shared memory, are not SharedPtrs.
  • Nothing existing changes: SharedArray::as_ptr / as_mut_ptr and DynamicSharedArray::get still return generic pointers.

Tests

  • shared_memory_helper runs its probe a second time through SharedPtr, with no features required:

    • the probe passes a non-inlined generic helper a Region<T> { base: SharedPtr<T>, offset };
    • it addresses the barrier and data slots with byte_add / cast;
    • it reports PASS shared-ptr: 9 launches, 36 CTA data/barrier checks.

    The raw *mut u8 helpers of the same example still fail MIR verification, as the README documents.

  • The new verify-code-shape.sh checks that shared_ptr_probe takes ptr addrspace(3) in the emitted LLVM IR, and that no PTX converts a generic address to a shared one (cvta.to.shared).

  • In a scratch kernel, not committed, the remaining API also computed correct values on the GPU:

    • SharedArray::shared_ptr, from_raw and a negative offset;
    • a SharedPtr passed to a closure;
    • addr_u32 differences.
  • A SharedPtr kernel parameter fails to compile with the exporter diagnostic above.

Validation

All runs used an RTX PRO 6000 (sm_120) with the CUDA 13.3 toolkit.

  • just fmt-check, clippy, test and test-cuda pass (3963 and 1505 tests).

  • just doc-check passes.

  • just check-guards passes every script except two, whose results could not be obtained here:

    • cargo deny --locked check stops at the advisory-database lock (NFS do not support locking). cargo deny check bans licenses sources passes for all three workspaces.
    • check-example-license-policy.sh hits the same lock.

    This PR adds no dependency and changes no lockfile.

  • Full scripts/smoketest.sh: 233/234 pass, the same as main. The one failure, sparse_mma_fp8, also fails on main.

  • shared_memory_helper passes on both backends: LLVM NVPTX, and libNVVM with --emit-nvvm-ir --arch=sm_120. Each backend runs both the direct and the SharedPtr probe. On libNVVM, the helper is still define i64 @…shared_ptr_probe…(ptr addrspace(3) %v0, i64 %v1).

Follow-ups tracked in #1337: #1336 moves cp_async_bulk_tensor_*_g2s_cta onto SharedPtr<u8> on top of this PR; SharedPtr forms for cp.async, ldmatrix/stmatrix and the tcgen05/wgmma descriptors; a cluster-shared counterpart from map_shared_rank.

🤖 Generated with Claude Code

…e(3)

`SharedPtr<T>` is a raw pointer into the issuing CTA's shared memory
that carries its address space in its type. It wraps a
`*mut SharedSlot<T>`, which the importer types as an `addrspace(3)`
pointer, as it already does for pointers to `SharedArray` and
`Barrier`, and the importer lowers `SharedPtr` itself to that pointer.
Parameters, returns, locals, struct fields and generic code of type
`SharedPtr<T>` are therefore CTA-shared pointers by type, across calls
to non-inlined helpers, with no dataflow inference.

- Constructors: `SharedArray::shared_ptr(&raw mut S)` and
  `DynamicSharedArray::shared_ptr()` / `shared_ptr_at(bytes)` keep the
  allocation's shared address; `unsafe SharedPtr::from_raw` is the one
  explicit locality assertion.
- `add` / `offset` / `byte_add` / `cast` keep the type; `as_ptr` widens
  to a generic pointer; `addr_u32` gives the 32-bit `.shared::cta`
  address for the `_shared_u32` forms.
- The importer drops projections onto `SharedPtr`'s field, which are
  the identity for this scalar-lowered newtype, so a `SharedPtr` nested
  in another place is read as a pointer value.
- A kernel entry cannot take a `SharedPtr`: the exporter already
  rejects `addrspace(3)` kernel parameters.
- `shared_memory_helper` runs its probe a second time through a
  non-inlined generic helper that takes the `SharedPtr` in a struct;
  `verify-code-shape.sh` checks the helper's `ptr addrspace(3)`
  parameter and that no PTX converts a generic address to shared.

Refs NVIDIA#1337, NVIDIA#1238.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Signed-off-by: Zihua Wu <[email protected]>
@nihalpasham nihalpasham added the cuda-oxide SIMT programming model: rustc backend, cargo oxide, cuda-device, cuda-host, book label Oct 11, 2026
@nihalpasham

Copy link
Copy Markdown
Collaborator

Landed through #1336 at 8bb24a4: your commit is the first one on that branch, unchanged and with your sign-off, plus the review follow-ups (SharedArray::shared_ptr made unsafe like as_raw_mut_ptr, the aggregate kernel-parameter refusal with its fixture, the shared_ptr_api regressions, the limits in the book). Closing this one as already fixed. Thanks for the type, it is what the .shared::cta form needed.

@nihalpasham nihalpasham added the already-fixed Resolved on a newer commit or branch; no action needed label Oct 20, 2026
nihalpasham added a commit that referenced this pull request Oct 27, 2026
…l TMA without SharedPtr (#1656)

## What This PR Does

This PR makes three changes to shared-memory addressing and CTA-local
TMA loads (following up on #1336, #1341, #1238, #1337, and #1249):

1. **Keep `.add()` and `.cast()` on shared pointers in `addrspace(3)`
(`mir-importer`)**
2. **Use 32-bit `i32` shared-memory offsets in `cuda-oxide-mlir-export`
(+ `#[grid_constant]` in `blockscale_gemm_cute`)**
3. **Remove `SharedPtr<T>` (`-3,141` lines) and take `dst: *mut u8`
directly in `cp_async_bulk_tensor_{1..5}d_g2s_cta` (+ route `ctm-rs` to
`.shared::cta`)**

---

### 1. Keep `.add()` and `.cast()` on Shared Pointers in `addrspace(3)`
(`mir-importer`)

Previously, `SlotAddrSpaceMap` only preserved `addrspace(3)` across
plain `_y = _x` copies. Calling `.add(offset)` or `.cast::<T>()` on a
shared pointer demoted it to a 64-bit generic pointer (`addrspace(0)`),
forcing an `addrspace(3) -> addrspace(0) -> addrspace(3)` round-trip:

```text
Rust kernel:
  let base = DynamicSharedArray::<u8, 1024>::get_raw();
  let tile = base.add(slot * TILE_BYTES).cast::<u32>();

BEFORE:
  base (ptr addrspace(3))
    ──► addrspacecast to generic (ptr)       <-- demoted on .add() / .cast()!
    ──► 64-bit generic GEP
    ──► addrspacecast back to shared (ptr addrspace(3))

AFTER:
  base (ptr addrspace(3))
    ──► GEP in ptr addrspace(3)              <-- stays in shared memory (0 casts)!
```

---

### 2. Use 32-bit `i32` Shared-Memory Offsets in
`cuda-oxide-mlir-export`

Shared memory (`!llvm.ptr<3>`) uses 32-bit addresses, but
`MirPtrOffsetOp` and SM120 `mma` / `epilogue` lowering in
`cuda-oxide-mlir-export` were extending 32-bit lane/warp indices to
64-bit `i64` before indexing `!llvm.ptr<3>`.

```text
BEFORE (64-bit i64 index math on 32-bit !llvm.ptr<3>):
  lane (i32) ──► zext to i64 ──► 64-bit swizzle math ──► GEP(!llvm.ptr<3>, i64)
  * Blinded LLVM's LoadStoreVectorizer (emitted 4 scalar 32-bit LDS instructions)
  * Left 20 64-bit SHF.R.U64 / IADD.64 instructions in the K-loop

AFTER (32-bit i32 index math on 32-bit !llvm.ptr<3>):
  lane (i32) ──────────────────► 32-bit swizzle math ──► GEP(!llvm.ptr<3>, i32)
  * LLVM's LoadStoreVectorizer automatically merges the 4 scalar loads into LDS.128!
  * Eliminates the 64-bit shift/add instructions in SASS
```

---

### 3. Remove `SharedPtr<T>` and Take `dst: *mut u8` in `.shared::cta`
TMA (#1336, #1341, #1238, #1337, #1249)

#### Why #1238 Happened on RTX 5090 (`sm_120a`)
Consumer Blackwell (`sm_120a`) has no hardware support for cross-block
TMA loads. When a kernel emits a **Cluster TMA** load
(`.shared::cluster`), `ptxas` generates an `if / else` with a software
emulation `CALL` in the `else` branch, which allocates an 8-byte stack
frame and causes `ptxas` to ignore `setmaxnreg` (warning `C7506`):

```text
Cluster TMA (`.shared::cluster`) on sm_120a:
  if (ptr.block_id == my_block_id) {
      UTMALDG.2D ;                                                 // fast hardware TMA
  } else {
      CALL __cuda_syscall_cp_async_bulk_tensor_2d_tile_unicast ;   // <-- kills setmaxnreg!
  }

Local Block TMA (`.shared::cta`) on sm_120a:
  UTMALDG.2D ;   // no branch, no CALL, 0 stack frame, keeps setmaxnreg!
```

#### Why We Don't Need `SharedPtr<T>` (Revising #1336 / #1341)
PR #1336 (and #1341) introduced `SharedPtr<T>` and lowered it to `ptr
addrspace(3)` across function signatures, return values, structs, and
arrays (`+3,141` lines of plumbing). That created a fundamental layout
mismatch:
* `rustc` always sizes `SharedPtr<T>` at **8 bytes** (`*mut T`).
* Under `p3:32:32:32` (`libNVVM` / `--nvptx-short-ptr`), LLVM sizes `ptr
addrspace(3)` at **4 bytes**, causing return-type mismatches in `llc`
and `misaligned address` crashes in arrays/structs.

With Change 1 (`SlotAddrSpaceMap`) + `convert_g2s_cta` (which already
accepts both `addrspace(3)` and `addrspace(0)` pointers, just like
`barrier: *mut Barrier`), `cp_async_bulk_tensor_{1..5}d_g2s_cta` works
directly with `dst: *mut u8`:

```rust
// Direct CTA-local TMA load with plain *mut u8 (0 cvta.to.shared, 0 stack frame):
cp_async_bulk_tensor_2d_g2s_cta(
    &raw mut TILE as *mut u8,
    tensor_map,
    tile_x,
    tile_y,
    &raw mut BAR,
);
```

We also updated `ctm_rs::tma::cp_async_bulk_tensor_shared_cta_global` to
call `cp_async_bulk_tensor_2d_g2s_cta` (`.shared::cta`) instead of
`cp_async_bulk_tensor_2d_g2s` (`.shared::cluster`), eliminating the
8-byte stack frame and runtime CTA-id check in `blockscale_gemm_ctm`.

---

## Verification

* **All 238 examples pass `scripts/smoketest.sh` on RTX 5090
(`sm_120a`)**, including `tma_copy/verify-code-shape.sh` (`.shared::cta`
emitted, `0` `cvta.to.shared`, `0` `shared::cluster`).
* **`just check`** (`fmt-check`, `clippy`, `test`, `test-cuda`,
`check-guards`, `check-intrinsics main`, `doc-check`): **all PASS**.
* **`scripts/check-primitives-api-compile.py`** (`api`, `fp16`,
`blockscale`): **all PASS**.
* **`setmaxnreg` + `.shared::cta` on `sm_120a`** (`ptxas -v` +
`nvdisasm`): `168` registers, `0` stack frame,
`USETMAXREG.DEALLOC.CTAPOOL 0x28`, `USETMAXREG.TRY_ALLOC.CTAPOOL UP0,
0xe8`, `UTMALDG.2D`, no `CALL`.
* **2048x2048x1024 MXFP4 Block-Scaled GEMM on RTX 5090 (Nsight `nsys
profile` kernel-active median of 31 launches)**:

| Kernel | Backend | Before This PR | **After This PR** | vs. Python
CUTLASS CTM (`10.880 us`) |

|:-------|:--------|:---------------|:------------------|:-------------------------------------|
| **Python CUTLASS CTM** (`reference/benchmark.py`) | `cutlass-mlir` |
`10.880 us` | **`10.880 us`** | baseline |
| **Rust `blockscale_gemm_cute`** | `cutlass-mlir` | `11.360 us` |
**`10.912 us`** | **`-0.032 us` (`-0.3%`)** |
| **Rust `blockscale_gemm_ctm`** | **Native `LLVM/PTX`** | `11.616 us` |
**`10.976 us`** | **`-0.096 us` (`-0.9%`)** |

---------

Signed-off-by: nihalpasham <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

already-fixed Resolved on a newer commit or branch; no action needed cuda-oxide SIMT programming model: rustc backend, cargo oxide, cuda-device, cuda-host, book

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants